Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Dockerfile #4068

Merged
merged 2 commits into from
Jan 13, 2025
Merged

Fix Dockerfile #4068

merged 2 commits into from
Jan 13, 2025

Conversation

wpoely86
Copy link
Contributor

The current Dockerfile doesn't build any more with 4.0.0 on my end. With these changes it works again.

johrstrom
johrstrom reviewed Jan 10, 2025
View reviewed changes
Dockerfile
@@ -23,6 +23,7 @@ RUN dnf -y update && \
patch \
lua-posix \
rsync \
xz \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this for sure? Isn't this the one that just had that vulnerability?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's the one. I'm also not a fan of it.

But it seems a requirement for building nokogiri: https://nokogiri.org/tutorials/installing_nokogiri.html#fedora-red-hat-and-centos

Or can we avoid building nokogiri?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As reference:

--------- IMPORTANT NOTICE ----------
Building Nokogiri with a packaged version of libxml2-2.11.7.
Configuration options: --host\=x86_64-redhat-linux --enable-static
--disable-shared
--libdir\=/opt/ood/ondemand/root/usr/share/gems/3.3/gems/nokogiri-1.15.6/ports/x86_64-linux/libxml2/2.11.7/lib
--with-iconv\=yes --disable-dependency-tracking --without-python
--without-readline --with-c14n --with-debug --with-threads --disable-shared
--enable-static CPPFLAGS\= CFLAGS\=-O2\ -U_FORTIFY_SOURCE\ -g\ -fPIC
The following patches are being applied:
  - 0001-Remove-script-macro-support.patch
  - 0002-Update-entities-to-remove-handling-of-ssi.patch
  - 0003-libxml2.la-is-in-top_builddir.patch
  - 0009-allow-wildcard-namespaces.patch
  - 0010-update-config.guess-and-config.sub-for-libxml2.patch
  - 0011-rip-out-libxml2-s-libc_single_threaded-support.patch

The Nokogiri maintainers intend to provide timely security updates, but if
this is a concern for you and want to use your OS/distro system library
instead, then abort this installation process and install nokogiri as
instructed at:

https://nokogiri.org/tutorials/installing_nokogiri.html#installing-using-standard-system-libraries

Note, however, that nokogiri cannot guarantee compatibility with every
version of libxml2 that may be provided by OS/package vendors.

Extracting libxml2-2.11.7.tar.xz into
tmp/x86_64-redhat-linux/ports/libxml2/2.11.7... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include=${opt-dir}/include
	--without-opt-include
	--with-opt-lib=${opt-dir}/lib64
	--without-opt-lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/usr/bin/$(RUBY_BASE_NAME)
	--help
	--clean
	--prevent-strip
	--enable-system-libraries
	--disable-system-libraries
	--use-system-libraries
	--enable-system-libraries
	--disable-system-libraries
	--use-system-libraries
	--enable-static
	--disable-static
	--enable-cross-build
	--disable-cross-build
	--enable-cross-build
	--disable-cross-build
	--with-zlib-dir
	--without-zlib-dir
	--with-zlib-include=${zlib-dir}/include
	--without-zlib-include
	--with-zlib-lib=${zlib-dir}/lib64
	--without-zlib-lib
	--with-xml2-source-dir
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:557:in
`xzcat_exe': xzcat not found (RuntimeError)
from
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:568:in
`tar_command'
from
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:579:in
`extract_file'
from
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:140:in
`block in extract'
from
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:138:in
`each'
from
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:138:in
`extract'
from
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/mini_portile2-2.8.8/lib/mini_portile2/mini_portile.rb:233:in
`cook'
	from extconf.rb:540:in `block (2 levels) in process_recipe'
	from extconf.rb:319:in `chdir'
	from extconf.rb:319:in `chdir_for_build'
	from extconf.rb:540:in `block in process_recipe'
	from <internal:kernel>:90:in `tap'
	from extconf.rb:438:in `process_recipe'
	from extconf.rb:872:in `<main>'

To see why this extension failed to compile, please check the mkmf.log which can
be found here:

/opt/ood/ondemand/root/usr/share/gems/3.3/extensions/x86_64-linux/3.3.0/nokogiri-1.15.6/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in
/opt/ood/ondemand/root/usr/share/gems/3.3/gems/nokogiri-1.15.6 for inspection.
Results logged to
/opt/ood/ondemand/root/usr/share/gems/3.3/extensions/x86_64-linux/3.3.0/nokogiri-1.15.6/gem_make.out

  /usr/share/rubygems/rubygems/ext/builder.rb:125:in `run'
  /usr/share/rubygems/rubygems/ext/ext_conf_builder.rb:28:in `build'
  /usr/share/rubygems/rubygems/ext/builder.rb:193:in `build_extension'
  /usr/share/rubygems/rubygems/ext/builder.rb:227:in `block in build_extensions'
  /usr/share/rubygems/rubygems/ext/builder.rb:224:in `each'
  /usr/share/rubygems/rubygems/ext/builder.rb:224:in `build_extensions'
  /usr/share/rubygems/rubygems/installer.rb:855:in `build_extensions'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/rubygems_gem_installer.rb:71:in
`build_extensions'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/rubygems_gem_installer.rb:28:in
`install'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/source/rubygems.rb:204:in
`install'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/installer/gem_installer.rb:54:in
`install'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/installer/parallel_installer.rb:186:in
`do_install'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/installer/parallel_installer.rb:177:in
`block in worker_pool'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/worker.rb:62:in
`apply_func'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/worker.rb:57:in
`block in process_queue'
  <internal:kernel>:187:in `loop'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/worker.rb:54:in
`process_queue'
/opt/ood/ondemand/root/usr/share/gems/3.3/ondemand/4.0.0-1/gems/bundler-2.3.6/lib/bundler/worker.rb:91:in
`block (2 levels) in create_threads'

An error occurred while installing nokogiri (1.15.6), and Bundler cannot
continue.

In Gemfile:
  ood_appkit was resolved to 2.1.4, which depends on
    rails was resolved to 7.0.8.5, which depends on
      actioncable was resolved to 7.0.8.5, which depends on
        actionpack was resolved to 7.0.8.5, which depends on
          actionview was resolved to 7.0.8.5, which depends on
            rails-dom-testing was resolved to 2.2.0, which depends on
              nokogiri
rake aborted!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it issue seems to be that nokogiri ship it's own version libxml2 in a tar.xz file...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you try with zlib-devel instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should likely just pull a package off of yum.osc.edu. Or expect a local package to exist somewhere.

Building is sort of the issue - but really it's installing. Installing from source is very flaky and error prone. We have a lot of steps in the packages (rpms and debs) that we can just utilize instead of doing it all over again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, but that means creating packages for all these which is not that trivial and has a large maintenance overhead.

It's the reason we run a container for ondemand in the first place ;)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, but that means creating packages for all these which is not that trivial and has a large maintenance overhead.

Not sure I follow what you mean by all these. We already build lots of packages. In fact those are the actual artifacts we're supporting. Container installations aren't really supported by us.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, now I'm a bit lost. What's the reason than for the bundle install and rake install in this Dockerfile?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, now I'm a bit lost. What's the reason than for the bundle install and rake install in this Dockerfile?

I think I see, we're getting tripped up on the word packages maybe? When I mean we build and support packages I mean .deb and .rpm files that we host on yum.osc.edu.

bundle and rake install install our dependencies (ruby gems) along with some nodejs packages.

Or .rpms for example pull in all our dependencies,

ondemand/packaging/rpm/ondemand.spec

Line 147 in 359e2aa

rake --trace -mj%{ncpus} build

But also do all sorts of other stuff like moving stuff around and creating directories and so on.

A Dockerfile that has to replicate all of these is bound to be error prone.

@johrstrom johrstrom self-requested a review January 13, 2025 15:17
johrstrom
johrstrom approved these changes Jan 13, 2025
View reviewed changes
Copy link
Contributor

@johrstrom johrstrom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fixes, I know we went through a lot, but this'll work until we actually fix up that Dockerfile.

@johrstrom johrstrom merged commit 7f0b1f0 into OSC:master Jan 13, 2025
22 checks passed
@wpoely86 wpoely86 deleted the docker branch January 13, 2025 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@johrstrom johrstrom johrstrom approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@wpoely86 @johrstrom